PCT 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
H04N 7/18 



Al 



(11) International Publication Number: WO 97/05744 

(43) International Publication Date: 13 February 1997 (13.02.97) 



(21) International Application Number: PCT/US967 12333 

(22) International Filing Date: 26 July 1996 (26.07.96) 



(30) Priority Data: 

08/508,057 



27 July 1995 (27.07.95) 



US 



(71) Applicant: SENSORMATIC ELECTRONICS CORPORA- 

TION [US/US]; 951 Yamato Road, Boca Raton. FL 33431- 
0700 (US). 

(72) Inventors: GLATT, Terry, Laurence; 131 SE. 9th Court, 

Boca Raton, FL 33060 (US). SCHIELTZ, Steven, W.; 
739 Camino Lake Circle, Boca Raton, FL 33486 (US). 
KUPERSMIT, Carl; 9021 Yearling Drive, Lake Worth, FL 
33467 (US). 

(74) Agent: TORRENTE, John, J.; Robin. Blecker, Daley & 
Driscoll, 330 Madison Avenue, New York, NY 10017 (US). 



(81) Designated States: AL, AM. AT, AU, AZ, BB, BG, BR, BY, 
CA, CH, CN, CZ, DE, DK, EE, ES, FI, GB, GE, HU, IL, 
IS, JP, KE, KG, KP, KR, KZ, LK, LR, LS, LT, LU, LV, 
MD, MG, MK, MN, MW, MX, NO, NZ, PL, PT, RO. RU, 
SD, SE, SG, SI, SK, TJ, TM, TR, TT, UA, UG, UZ, VN, 
ARIPO patent (KE, LS, MW, SD, SZ, UG), Eurasian patent 
(AM, AZ, BY, KG, KZ, MD, RU. TJ, TM), European patent 
(AT, BE, CH, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, 
MC, NL, PT, SE), OAPI patent (BF, BJ, CF, CG, CI, CM, 
GA, GN, ML, MR, NE, SN, TD, TG). 



Published 

With international search report. 
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(57) Abstract 

A video surveillance system (10) has a camera equipped with a fisheye 
lens (20) having a substantially hemispheric field of view. The system (10) 
implements operations equivalent to pan. tilt, and zoom of a conventional 
camera. The camera produces a distorted fisheye image due to the properties 
of the fisheye lens (20). The system (10) corrects the distortion by mapping the 
pixels of the fisheye image to coordinates produced by selecting a particular 
part of the fisheye image to be viewed. The fisheye image formed by the 
camera is split into four separate image components (15. 16, 17, 18) carried by 
four bundles of optical fibers (35, 36. 37, 38). Each bundle has a CCD (45. 46, 
47, 48) and associated image processing circuitry (65, 66, 67, 68) which forms 
an electronic representation of the image component carried by that bundle. 
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IMAGE SPLITTING/ FORKING AND PROCESSING DEVICE AND METHOD 
FOR USE WITH NO MOVING PARTS CAMERA 

Field of the Invention ; 

This invention relates generally to the field of video 
surveillance systems. More specifically, it relates to an 
image forming and processing device including a fisheye lens 
having a substantially hemispherical field of view. The 
invention allows an operator to view a selected part of the 
image formed by the fisheye lens as if it were formed by a 
normal lens by simulating the panning, tilting or zooming of 
the normal lens. This allows the operations of panning, 
tilting and zooming to be implemented without the use of 
moving parts. 
15 Description of Related Art ; 

Surveillance cameras are commonly used to monitor areas 
of retail stores, factories, airports and the like. In order 
to use a single camera to survey a large area, the camera is 
typically provided with mechanisms to enable it to pan, tilt 
20 and zoom. Such mechanisms increase the complexity and hence 
the cost of the camera and can also adversely affect its 
reliability. Due to the presence of moving parts, mechanical 
pan, tilt and zoom devices are subject to damage and 
degradation brought on by extremes of temperature, moisture 
25 and dust. In addition, such mechanical systems consume 
relatively large amounts of power. A surveillance camera 
capable of panning, tilting and zooming without the use of 
moving parts would therefore provide significant advantages 
over existing surveillance cameras. 
30 in U.S. Patent No. 5,185,667, Zimmermann proposes such 

a camera having no moving parts. In the device specified in 
that patent, a fisheye lens is coupled to a video camera such 
that the camera produces an electronic image. Due to the 
characteristics of the fisheye lens, the image is distorted. 
35 The distortion in the image is corrected by means of an 
algorithm. 

One of the limitations of the system proposed by 
Zimmermann is that the camera is unable to provide sufficient 
resolution for effective zooming. Since a fisheye lens 
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renders a distorted image of an entire hemisphere, parts of 
the image, especially at its peripheries are distorted. The 
image is formed on a change coupled device (CCD) having a 
limited number of pixels. In order to view the image as a 
5 normal (non-distorted) image, it is necessary to transform 

the image electronically. The limited number of pixels in k 
the CCD causes the transformed image to be poorly resolved. 4 
In order to provide acceptable resolution, a CCD made of 
approximately 156,000,000 would be needed. 

10 The best available CCD's have approximately 16 , 000,000 

pixels (4,000 x 4,000) and operate at clocking rates of the 
order of 10 Mhz. However, in order to satisfy the NTSC 
sampling rate of 30 samples per second, a clocking rate of 
480 MHz is needed. Thus, the type of resolution required for 

15 an NTSC picture with the desired magnification cannot be 
achieved using the prior art. 

In U.S. Patent No. 5,200,818, Neta et al. describe a 
system in which a wide angle scene is monitored by means of 
a plurality of sensor arrays mounted on a generally 

2 0 hemispherical surface. Each sensor array has its own lens 

system. This allows a wide field to be monitored without the 
need for moving parts to effect panning and tilting. The 
resolution of the system would be relatively high due to the 
plurality of sensor arrays. However a system such as that 
25 described by Neta et al. would be very costly to implement 
due to the large number of high quality components needed. 

It is an object of the present invention to provide a 
surveillance camera apparatus, having a substantially 
hemispherical field of view and capable of effecting the 

3 0 operations of panning, zooming and tilting without the use of 

moving parts, while still providing sufficient resolution to 
allow the desired magnification. 

It is a further object of the invention to provide a 
surveillance camera apparatus, having a substantially ' 
3 5 hemispherical field which allows an operator to view parts of * 
the field of view as if they were acquired by a camera having 
a conventional lens and being capable of panning, tilting and 
zooming. 
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These and other advantages are achieved by the invention 
described herein. 

SUMMARY OF THE INVENTION 
The present invention is a an image forming and 
5 processing device for use with a video camera. The device 
comprises a lens having a wide field of view (preferably a 
f isheye lens) . The lens forms a first image having a 
distortion caused by the lens. An image splitter splits the 
first image into a plurality of images. At least one image 
10 sensor is provided for converting at least one of the 
plurality of images into an electronic representation. A 
processor corrects the distortion so that at least part of 
the first image can be viewed substantially without the 
distortion. The image splitter preferably comprises a 
15 plurality of bundles optical fibers, each bundle of optical 
fibers transmitting a part of the first image. The image 
sensor preferably comprises a CCD connected to at least one 
of the bundles of optical fibers for forming an optical 
representation of the part of the first image transmitted by 
20 that bundle of optical fibers. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a block diagram of a system embodying the 
invention; 

Fig. 2A is a plan view of the image plane of the f isheye 
25 lens showing a distorted f isheye image; 

Fig. 2B is a diagram of a selected part of the f isheye 
image, corrected using the present invention; 

Fig. 3 is a perspective view of the image splitter of 
the invention; 

30 Fig. 4 is a perspective view of the fiber optic bundles 

in the image splitter; 

Fig. 5 is a block diagram of the f isheye distortion 
correction system of the invention; 

Fig. 6 is a diagram showing the projection of a point C 
35 at tilt angle b on the Y axis of the image plane as a result 
of the f isheye distortion; 

Fig. 7 is a diagram of the image plane X-Y showing the 
projection of a point C on the image plane; and 
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Fig. 8 is a three dimensional diagram showing the 
primary axis of the fisheye lens, the primary axis of a 
hypothetical camera panned and tilted to point at point C. 

DETAILED DESCRIPTION 
5 The following is a description of the preferred 

embodiment of the present invention. It is intended to be 
illustrative of the invention and not limiting. The full 
scope of the invention is to be determined by the appended 
claims and their equivalents. 

10 The invention is shown in block diagram form in Fig. 1. 

Typically the invention is used in the surveillance of 
premises such as warehouses, stores, bus or train stations 
and the like. To this end, system 10 is provided with a lens 
20 which has a substantially hemispherical field of view, for 

15 example a fisheye lens. It is preferable to have an 
azimuthal view of 180°, a zenithal view of 90* and an 
infinite depth of field. This produces the desired 
substantially hemispherical field. The preferred lens is a 
commercially available equidistant fisheye lens having a 

20 focal length of 1.9 mm, and an f stop of 1.8. Lens 20 has a 
primary axis Z and forms a circular image 14 on image plane 
13. 

Due to the properties of lens 20, image 14 is distorted. 
Specifically, the orientation of objects in image 14 is 

25 altered relative to their real orientation. For example, an 
object 11 in the field of view of lens 20 (See Fig. 8) will 
appear on the periphery of image 14 in distorted form as 
shown in Fig. 2. 

Image 14 is preferably split into four separate 

30 components by splitter 30. Image 14 could be split into any 
number of components, depending on the resolution required 
and the available technology. When image 14 is split into 
four components, each component respectively contains an 
image 15, 16, 17 or 18 made up of one quadrant of circular 

35 image 14. (See Fig. 2). Splitter 30 is made up of four 
light conduits 25, *26, 27 and 28. Light conduits 25, 26, 27 
and 28 respectively contain coherent fiber optic bundles 35, 
36, 37 and 38 (See Fig. 4). Images 15, 16, 17 and 18 are 
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thus respectively carried in conduits 25, 26, 27 and 28 by 
fiber optic bundles 35 , 36, 37 and 38. 

Splitter 30 is shown in greater detail in Figs. 3 and 4. 
Splitter 30 is made up of a housing 32 to which are attached 
5 conduits 25, 26, 27 and 28. Optical fiber bundles 35, 36, 37 
and 38 housed in conduits 25, 26, 27 and 28 respectively, 
branch off from a major bundle of fibers, terminating at 
image plane 13 in a polished surface. See Fig. 4. Optical 
fiber bundles 35, 36, 37 and 38, are each made up of a 

10 plurality of optical fibers. Each optical fiber carries a 
sample of image 14 formed by fisheye lens 20 and has a 
diameter of approximately 10 /xm. 

Images 15, 16, 17 and 18 respectively travel along each 
of conduits 25, 26, 27 and 28 and impinge respectively upon 

15 sensors 45, 46, 47 and 48. Sensors 45, 46, 47 and 48 are 768 
x 480 CCD's with fiberoptic windows formed from a fiberoptic 
faceplate which allows for direct coupling of the CCD's to 
the optical fibers. Suitable fiberoptic faceplates are 
available from Galileo Electro-optics Corporation of 

20 Sturbridge, Massachusetts under the name "CP Series." Images 
15, 16, 17 and 18 are respectively converted by the sensors 
into representative electrical signals 55, 56, 57 and 58. 

Signals 55, 56, 57 and 58 are fed into CCD control 
processor 60 which is made up four identical off the shelf 

25 video camera sensor image controllers 65, 66, 67 and 68, each 
corresponding respectively to one of signals 55, 56, 57 or 
58. Each of the control processors contains a CCD clocking 
circuit 72, a video processing circuit 74 and a color space 
converter 76. Color space conversion circuit 76 produces 

30 chrominance and luminance signals Cr, Cb and Y for each 
signal 55, 56, 57 and 58. 

Control processors 65, 66, 67 and 68 respectively 
produce video outputs 85, 86, 87 and 88 in the form of 
luminance and chrominance components suitable for compression 

35 by encoder 100. Compression of the video signals 85, 86, 87 
and 88 allows a very large number of image samples to be 
transmitted over a channel having limited bandwidth. The 
video outputs are therefore compressed if the lens is at a 
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location remote from correction circuit 140. Encoder 100 
compresses the video signals 85 , 86 , 87 and 88 by compressing 
them in accordance with a compression scheme, for example, 
MPEG or H. 261. Alternatively, a sub-band coding scheme can 
5 be used. Encoder 100 packetizes the video signals into a 
serial data stream for transmission over high speed network 
110 such as coaxial cable or optical fibers. The compressed 
video signals are received by decoder 120 which performs a 
transform on the compressed video signals which is the 

10 inverse of the transform performed by encoder 100. 

Decoder 120 produces a decoded video signal 130 which is 
fed into correction circuit 140. The purpose of correction 
circuit 140 is to correct the distortion introduced by 
fisheye lens 20. This correction is performed in accordance 

15 with the algorithm described below. Correction circuit 140 
produces a corrected signal 150 which is displayed on display 
160. 

The following is a description of the system for 
correcting the fisheye distortion of image 14. For the sake 

20 of simplicity, it will be assumed that the entire fisheye 
image 14 is formed on the surface of a single CCD 180 and 
that splitter 30 is not used. CCD 180 has axes X and Y. 
Lens 20 is mounted at a mounting point 17 vertically above 
surveillance plane 19, preferably such that principal axis Z 

25 is perpendicular to surveillance plane 19. Surveillance 
plane 19 is the floor of a room 15. Mounting point 17 is on 
the ceiling of room 15. Axes X, Y and Z intersect at center 
point I on the surface of CCD 180. The surface of CCD 180 
forms image plane 13 which is parallel to surveillance plane 

30 19. 

Mounting the camera and fisheye lens above the 
surveillance field (i.e. on ceiling rather than on a wall) 
has several advantages. Firstly, with the camera on the 
ceiling, the field of view covers a full 360°. This allows 
35 the simulation of a pan through 360° rather than a pan range 
limited by the presence of the wall. In the case of a 
ceiling mounted lens, the hypothetical (simulated) pan axis 
is the primary axis Z of the fisheye lens, rather than an 
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axis perpendicular to the primary axis in the case of a wall 
mounted lens. The angle about the primary axis Z is 
maintained from the object to the image. This facilitates 
the calculation of radial coordinates because the pan axis is 
5 already in radial form and no conversion is needed. 

When any object is viewed on monitor 240, the vertical 
center line of the image intersects the center point I of the 
image plane. The primary axis Z of the lens passes through 
this center point. There is therefore no need to rotate the 

10 images to view them in their correct orientation. In the 
correction algorithm set forth in U.S. Patent No. 5,185,667, 
rotation of the image is separately calculated. Such a 
separate operation is not needed with the present invention. 
When the lens is placed on a wall, objects of interest,. 

15 and objects which are furthest away tend to be at the center 
of the fisheye image. The greatest resolution is needed to 
view the details of those objects. When the fisheye lens is 
placed vertically above the surveillance plane, objects in 
the center are usually closest to the lens. Viewing of such 

20 objects does not require high resolution and those objects 
are the least distorted. Objects which are furthest away 
from the lens appear at the peripheries of the fisheye image. 
However, the image formed by a fisheye lens has a higher 
density and therefore a lower CCD image resolution at the 

25 center than at . its peripheries. Consider a part of a 
fisheye image having a radius of "R." The density of the 
pixels in the CCD on which the image is formed is uniform. 
Along a line passing through the center of the CCD, the image 
is spread over 2R pixels. At the circumference of the 

30 image, the image is spread over 7rR (half the circumference) - 
7T/2 more pixels than for objects appearing at the center of 
the image. Thus placing the lens vertically above the 
surveillance plane provides far better resolution for distant 
objects than if the lens is placed perpendicular to the 

35 surveillance plane. 

The following description refers to Fig. 5. 
Fisheye lens 20 has a 180 degree field of view covering area 
"A." With lens 20 is mounted on the ceiling of room 15, area 
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A includes the floor and walls of the room, Fisheye lens 20 
forms a fisheye image A d of area A on image plane 13. Any 
point in area A represented by unique coordinates (x;y) , is 
displaced to point (x d ;y d ) in the fisheye image A d in 
5 accordance with the characteristics of fisheye lens 20 . 
Image plane 13 (the surface of CCD 180) is made up of a 
matrix comprising a plurality of pixels 182. Each pixel has 
unique fisheye coordinates. CCD thus produces an electronic 
representation of area A. This representation is fed into 

10 CCD control processor 250 (identical to control processor 60) 
which produces chrominance and luminance values for each 
pixel in CCD 180. Those chrominance and luminance values are 
stored in dual ported image memory ("DPIM") 200. The present 
invention allows the user to manipulate the fisheye image 

15 electronically in order to implement the operations of 
panning, tilting and zooming. Thus a sub-area a of area A 
can be examined in detail by the transformation of sub-area 
ot d of area A d from a distorted fisheye image into a normal 
image. 

20 When the system is powered up a default corrected sub- 

area a c appears on monitor 240. The user selects sub-area a 
by means of area select unit 210 - a control station having 
a keyboard and a pointing device. This is done by using 
pointing device 214 to simulate the panning and a tilting of 

25 a hypothetical conventional camera. The image on monitor 24 0 
appears to have been formed by a conventional camera. In 
reality, it is formed by correction of part of fisheye image 
14. The selection of sub-area a provides the normal (non- 
fisheye) coordinates of an object in the center of sub-area 

30 a. This operation simulates the pointing of the primary axis 
(IC in Fig. 8) of hypothetical conventional camera at the 
object. The hypothetical camera is mounted at mounting point 
17 with its primary axis IC passing through center point I 
and through the center of sub-area a. Pointing this 

35 hypothetical camera by means of input device 214 such that a 
sub-area a appears on monitor 24 0 also causes area select 
unit 210 to generate the pan and tilt angles which would be 
associated with the hypothetical camera positioned at 
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hypothetical pan and tilt angles so that it points at an 
object in sub-area a* 

When the user selects sub-area a the system 
automatically converts a d (the distorted f isheye image of area 
5 a) into a corrected image a c . This allows the user to view 
the sub-area a on monitor 240 as if it were formed by the 
hypothetical (non-f isheye) camera which had been panned and 
tilted to point at sub-area a. 

Each of the pixels in the fisheye image A d is stored at 

10 a unique address in DPIM 200 in the form of the intensity and 
color data generated by CCD 180 via control processor 250. 
DPIM 200 thus contains a digital electronic representation of 
the distorted fisheye image A d of area A. For any sub-area 
a of area A, DPIM 200 contains an electronic representation 

15 of the corresponding distorted sub-area a d . 

Image plane 13 is the plane formed by the X and Y axes 
as shown in Figs. 6, 7 and 8. Primary axis Z of lens 20 is 
perpendicular to the X and Y axes. If a user wished to view 
in detail the scene centered around point C (i.e sub-area q- 

20 the image shown in Fig. 2B) with a hypothetical non-f isheye 
lensed camera, the user would instruct the camera to tilt by 
an angle b relative to the primary axis Z. Doing so would 
orient the hypothetical camera such that the hypothetical 
primary axis (center line IC) passes through the center point 

25 I of image plane 13 and through point C. 

Had it been captured by the hypothetical conventional 
camera, area a would appear on CCD 180 as an image 3 00 
centered at line 32 0 and made up of a large number of 
horizontal lines of pixels 310. (See Fig. 2A) . Each pixel 

30 on a particular horizontal line is displaced from center line 
320 by a particular distance x. That distance corresponds to 
an angle "a" relative to center line IC (See Fig. 8) or angle 
a 1 about primary axis Z . 

Each pixel in image 14 can be described by reference to 

35 a set of rectangular or polar coordinates. Thus, referring 
to Figs. 7 and 8, the pixel at point C on center line IC can 
be located by reference to polar coordinates in the form of 
tilt angle b (See Fig. 6) and angle a - the displacement of 
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the pixel from center (for point C r a is equal to zero since 
C lies on the X axis) . Similarly, moving along a horizontal 
line in CCD 180 (i.e., moving parallel to the Y axis), a 
pixel at point S can be described by reference to tilt angle 
5 b 1 relative to principle axis Z and pan angle a 1 relative to 
center line IC. The corresponding rectangular coordinates 
are x d and y d . 

Referring again to Fig. 2A, it can be seen that due to 
the nature of the fisheye lens, the fisheye image is 

10 distorted. Objects located close to the principal axis of 
fisheye lens 20 appear on CCD 180 substantially normally (See 
area 182) , whereas, objects further from the principal axis 
are progressively more distorted (See area 184). The 
information carried by a pixel located at point (x;y) in a 

15 non-fisheye image will, in the fisheye image, be located at 
( x d ; yd) ' whe ^e (x d ;y d ) are displaced from (x;y) by an amount 
dependent on the properties of fisheye lens 20. 

It is a fundamental property of a fisheye lens that the 
image of a point located at an angle of rotation b* relative 

20 to the primary axis will be projected on the image plane a 
radius r from the primary axis in accordance with the 
formula: 

r = f.b' 

where r is the distance from center point I; 
25 f is a lens constant in mm/radian indicative of the 

distortion caused by the fisheye lens; and 

b 1 is the angle of an incident ray from an object to the 
primary axis (in radians) . 

It is also a fundamental property of a fisheye lens that 
30 the angle from a point in the field of view to its projection 
on the image plane is maintained. 

These two properties are used to derive a new formula 
which allows selected parts of the fisheye image to be viewed 
as if they were formed by a conventional camera panned, 
35 tilted or zoomed in on an area of interest in the field of 
view. This formula relates the pan and tilt angles of a 
hypothetical camera described above to the rectangular 
coordinates of a corrected image. The following is a 
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description of how that formula is derived and applied to 
achieve the objects of the invention. 

From Fig. 6 it can be seen that a point C located at a 
tilt angle b relative to the principal axis of the lens forms 
5 an image on image plane IP at a radius r=r c from center point 
I. As stated above, for a particular fisheye lens, the 
relationship between tilt angle b and the radius at which the 
image of point C forms is: 

r=f .b (1) 

10 In Fig. 8, point C lies in the plane formed by the Y and 

Z axes and at a tilt angle of b relative to the primary axis 
Z. The line IC from the center I of the image plane to point 
C is taken as the primary axis of a hypothetical camera lens 
pointed at point C. Line CS extends from point C to a point 

15 S. CS is parallel to the X axis. CS thus represents a 
horizontal line of pixels in CCD 180. Consider a pixel at 
S, at a particular radius r from I, the center of the CCD, 
and at a pan angle "a'" about the primary axis of the 
hypothetical camera lens and at a tilt angle b' relative to 

20 the primary axis of the fisheye lens. The rectangular 
coordinates of that pixel are: 

X=f .b' .cos (a' ) (2) 

Y=f .b' .sin(a f ) ( 3) 

Equations (2) and (3) convert the polar coordinates of 

25 any particular pixel of the fisheye image formed on CCD to 
rectangular coordinates. The pixel at point S can therefore 
be located by reference to tilt angle b' (an angle measured 
off the principal axis Z) and pan angle a' (the angle of 
rotation around the principal axis Z) . 

3 0 When the system powers up a default area a is displayed, 

corresponding to the initial area at which the hypothetical 
camera is pointing. For convenience, this area lies along 
the primary axis Z (so the tilt angle b is zero). The pan 
angle is also zero (i.e., line IC lies along the X axis). 

35 The hypothetical camera (with the primary axis of its lens 
lying along line IC) is then tilted by an angle of "b" 
relative to the primary axis Z of the fisheye lens so that it 
points at an object centered at point C. In order to make 

11 
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the operation of the correction system transparent to the 
user, the panning and tilting of the hypothetical camera is 
measured relative to the initial position of the hypothetical 
camera. Thus, the position of a pixel representing a point 
5 at S will be expressed in terms of tilt angle "b" and the 
angle of point S from center line IC - angle M a" the amount 
of pan from center line IC to point S. 

The following is a description of the manner in which 
the position of a pixel representing point S in the fisheye 
10 image can be described by reference to angle a - its 
displacement from the center line IC and angle b - the tilt 
angle of a hypothetical normal camera panned and tilted so 
that it's principal axis is aligned with point C. 
Referring to Fig. 8, it is seen that 
15 tan (a 1 ) = SC/PC 

SC=IS.sin(a) 
PC=IC.sin(b) 
IC-IS.cos(a) 
therefore tan(a') = IS.sin(a)/IS.cos(a) .sin(b) 
20 = tan(a)/sin(b) 

a ' = tan* 1 (tan (a) /sin (b) ) (4) 

cos(b') = IP/IS 
IP=IC.cos(b) 
IC=IS . cos (a) 
25 therefore cos(b') = IS . cos (a) .cos (b) /IS 

= cos(a) .cos(b) 

b' = cos" 1 (cos(a) .cos(b) ) (5) 

From equations (2) and (3), for a given fisheye lens, 
X d =fb l cos(a' ) and Y d =fb 1 sin(a ■ ) . Substituting the values of 
3 0 a 1 and b* from equations (4) and (5) into equations (2) and 
(3): 

X d = f. cos" 1 (cos (a) .cos(b)) .cos (tan" 1 (tan (a) /sin (b) ) ) . . . (6) 
Y d = f .cos' 1 (cos(a) .cos(b)) .sin (tan' 1 (tan(a)/sin(b) ) ) ... (7) 
These formulas allow the coordinates of the pixels 
35 centered around center line IC to be calculated simply from 
knowledge of angular coordinates in the form of the tilt 
angle "b" of a hypothetical camera (a measure of the distance 
of the point from the center of the fisheye image) and the 

12 



WO 97/05744 



PCT/US96/12333 



angle "a" of a pixel relative to center line IC. This 
formula provides a very simple means for effectuating 
panning, tilting and zooming from the fisheye image* 

To effect panning of the hypothetical camera, pan angle 
5 p is added to angle a 1 to form new angle a". Thus, a"= p + 
a' . 

Substituting this into equation (4) gives: 

a"= p + tan" 1 (tan(a)/sin(b)) (8) 

Substituting equation (a) into equations (6) and (7) : 
10 X d =f .cos" 1 (cos(a) .cos(b) ) .cos(p + tan 1 (tan(a)/sin(b) ) 

(9) 

Y d =f .cos" 1 (cos(a) .cos(b) ) .sin(p + tan 1 (tan (a) /sin (b) ) 
(10) 

As pointing device 214 is moved to simulate panning 

15 and/or tilting of the hypothetical camera, the rectangular 
coordinates (X;Y) of each pixel in each line of pixels in 
sub-area a are generated by area select unit 210 and stored 
in look-up table ("LUT") 222. The system also automatically 
calculates the coordinates (X d ;Y d ) of the fisheye image from 

20 the using equations (9) and (10). For each set of normal 
coordinates (X;Y) in sub-area a, the calculated coordinates 
(X d ;Y d ) are stored in LUT 222 as addresses in DPIM 200. 

All of the coordinates for the fisheye image could be 
pre-calculated or only the coordinates for a particular area 

25 need be calculated as the area is selected. In either case, 
the coordinates are stored in LUT 222 and the corresponding 
pixels are stored in DPIM 200. This allows the pixels 
corresponding to those calculated coordinates to be fetched 
from CCD 180. The fetched pixels are then displayed on 

30 monitor 240 at locations (X;Y) just as if the image had been 
formed by the panning and tilting of a normal camera to 
coordinates (X;Y) . 

Zooming can be accommodated by varying the amount that 
angle a i incremented between pixels and the amount b is 

35 incremented between rows when calculating the contents of LUT 
222. For example, if there are 400 pixels on a horizontal 
display line and a is incremented from -20° for the left side 
of the display in steps of .1°, a 40* horizontal field of 
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view will result. Likewise, to display a 30° vertical field 
of view that would correctly maintain the 4 : 3 aspect ratio of 
a standard display, the 483 display lines would be obtained 
by incrementing b by .062° between each horizontal display 
5 line. 

The contents of LUT 222 and DPIM 200 are represented in 
the following table: 

TABLE I 



15 



30 



ADDRESS SEQUENCE 
FOR BOTH DATA 
STRUCTURES 


FEA GENERATOR LUT 
CONTENTS 


DUAL PORT MEMORY 
CONTENTS 


Starting Address 


Address of 1st 
pixel of 1st row 


1st pixel 1st row 


Starting Address + 
1 


Add. of 2nd pixel 
of 1st row 


2nd pixel 1st row 


• 


• 


• 


• 


• 


• 


• 


• 


• 


Starting Address + 
H 


Add. of 1st pixel 
of 2nd row 


1st pixel 2nd row 


Starting Address + 
H + 1 


Add. of 2nd pixel 
of 2nd row 


2nd pixel 2nd row 


• 


• 


• 


• 


• 


• 


• 


• 


• 


Starting Address + 
2H 


Add. of 1st pixel 
of 3rd row 


1st pixel 3rd row 


Starting Address + 
2H + 1 


Add. of 2nd pixel 
of 3rd row 


2nd pixel 3rd row 


• 


• 


• 


• 


• 


• 


« 


• 


• 



H = Number of pixels per line in display processor. 

By retaining multiple images in DPIM 200, a historical 
35 log of images over time can also be stored. The oldest image 
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is continually overwritten with the current image as the 
memory capacity is exceeded, thus maintaining a revolving log 
of images generated over a predetermined time period. Thus, 
by appropriate selection of an address in DPIM 200 by fisheye 
address generator, images captured in the preceding 
predetermined time interval can be displayed when an alarm 
event occurs (e.g. an intruder attempting to enter the 
monitored premises and triggering a sensor) . 

Using a 360 degree image, this system implements the 
operations of panning and tilting without any moving parts. 

This increases the reliability of the camera while limiting 
the cost of acquiring and maintaining it. The invention thus 
enables the monitoring of a large area by means of a single 
camera covering a wide field of view. 
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WE CLAIM ; 

1. An image forming and processing device for use with 
a video camera , comprising: 

a lens having a wide field of view, the lens 
5 forming a first image having a distortion caused by the lens; 

an image splitter for splitting the first image into a 
plurality of images; 

at least one image sensor, for converting at least 
part of one of the plurality of images into an electronic 
10 representation; 

a processor for correcting the distortion so that 
at least part of one of the plurality of images can be viewed 
substantially without the distortion. 

2. The image forming and processing device of claim 1 
15 wherein the image splitter comprises a plurality of bundles 

optical fibers, each bundle of optical fibers transmitting a 
part of the first image. 

3. The image forming and processing device of claim 2 
wherein the image sensor comprises a CCD connected to at 

20 least one of the bundles of optical fibers for forming an 
optical representation of the part of the first image 
transmitted by that bundle of optical fibers. 

4. The image forming and processing device of claim 1 
wherein the image splitter divides the first image into 

25 quadrants. 

5. The image forming and processing device of claim 1 
wherein the lens is a fisheye lens. 

6. The image forming and processing device of claim 1 
further comprising compression means for compressing the 

30 electronic representation to form a compressed electronic 
representation . 

7. The image forming and processing device of claim 1 
wherein the image splitter comprises a plurality of image 

16 



WO 97/05744 PCT/US96/12333 

conduits, each of the image conduits carrying one of the 
plurality of images. 

8. The image forming and processing device of claim 1 
wherein the processor is adapted to perform a transformation 

5 on the electronic representation, the transformation being 
equivalent to panning a camera. 

9. The image forming and processing device of claim 1 
wherein the processor is adapted to perform a transformation 
on the electronic representation, the transformation being 

10 equivalent to tilting a camera. 

10. The image forming and processing device of claim 1 
wherein the processor is adapted to perform a transformation 
on the electronic representation, the transformation being 
equivalent to zooming a lens. 

15 11. A method of monitoring an area, the method 

comprising the steps of: 

forming an optical image of substantially the 
entire area by means of a fisheye lens having a wide field of 
view, such that the image has a distortion caused by the 
20 fisheye lens; 

splitting the optical image into a plurality of 
sub- images ; 

converting at least part of one of the sub-images 
into an electronic representation; 

25 processing the electronic representation, thereby 

correcting the distortion. 

12. The method of claim 11 further comprising the step 
of compressing the electronic representation to form an 
encoded electronic representation. 

30 13. The method of claim 12 wherein the step of 

compression the electronic representation is performed prior 
to the step of processing. 

14. The method of claim 11 further comprising the step 
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of performing a transformation on the processed electronic 
representation equivalent to panning a camera. 

15. The method of claim 11 further comprising the step 
of performing a transformation on the processed electronic 

5 representation equivalent to tilting a camera. 

16. The method of claim 11 further comprising the step 
of performing a transformation on the processed electronic 
representation equivalent to tilting a lens. 

17. The method of claim 11 further comprising the step 
10 of performing a transformation on the processed electronic 

representation equivalent to zooming a lens. 
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